{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding a variable to capture NA\n",
    "\n",
    "In the previous lectures within this section we studied how to replace missing values by the most frequent category or by extracting a random sample of the variable. These 2 methods assume that the missing data are missing completely at random (MCAR), and are suitable when the number of missing data is small, otherwise it may affect the distribution of the target within the labels of the variable.\n",
    "\n",
    "So what if the missing data are not small or not MCAR?\n",
    "\n",
    "We can capture the importance of missingness by creating an additional variable indicating whether the data was missing for that observation (1) or not (0). The additional variable is a binary variable: it takes only the values 0 and 1, 0 indicating that a value was present for that observation, and 1 indicating that the value was missing for that observation.\n",
    "\n",
    "The procedure is exactly the same as for numerical variables.\n",
    "\n",
    "\n",
    "### Advantages\n",
    "\n",
    "- Easy to implement\n",
    "- Captures the importance of missingess if there is one\n",
    "\n",
    "### Disadvantages\n",
    "\n",
    "- Expands the feature space\n",
    "\n",
    "This method of imputation will add 1 variable per variable in the dataset with missing values. So if a dataset contains 10 features, and all of them have missing values, we will end up with a dataset with 20 features. The original features where we replaced the NA by the frequent label or random sampling, and additional 10 features, indicating for each of the variables, whether the value was missing or not.\n",
    "\n",
    "This may not be a problem in datasets with tens to a few hundreds of variables, but if your original dataset contains thousands of variables, by creating an additional variable to indicate NA, you will end up with very big datasets.\n",
    "\n",
    "In addition, data tends to be missing for the same observation on multiple variables, so it may also be the case, that many of your added variables will be actually similar to each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "===============================================================================\n",
    "\n",
    "## Real Life example: \n",
    "\n",
    "### Predicting Sale Price of Houses\n",
    "\n",
    "The problem at hand aims to predict the final sale price of homes based on different explanatory variables describing aspects of residential homes. Predicting house prices is useful to identify fruitful investments, or to determine whether the price advertised for a house is over or underestimated, before making a buying judgment.\n",
    "\n",
    "=============================================================================\n",
    "\n",
    "In the following cells, I will demonstrate NA imputation by random sampling + adding an additional variable using the House Price datasets from Kaggle.\n",
    "\n",
    "If you haven't downloaded the datasets yet, in the lecture \"Guide to setting up your computer\" in section 1, you can find the details on how to do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "% matplotlib inline\n",
    "\n",
    "# for regression problems\n",
    "from sklearn.linear_model import LinearRegression, Ridge\n",
    "\n",
    "# to split and standarize the datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# to evaluate regression models\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### House Price dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "SalePrice      0.000000\n",
       "BsmtQual       0.025342\n",
       "GarageType     0.055479\n",
       "FireplaceQu    0.472603\n",
       "dtype: float64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's load the dataset with a few columns for the demonstration\n",
    "cols_to_use = ['BsmtQual', 'FireplaceQu', 'GarageType', 'SalePrice']\n",
    "\n",
    "data = pd.read_csv('houseprice.csv', usecols=cols_to_use)\n",
    "\n",
    "# let's inspect the percentage of missing values in each variable\n",
    "data.isnull().mean().sort_values(ascending=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**To evaluate whether adding this additional variables to indicate missingness improves the performance of the ML algorithms, I will replace missing values by random sampling (see previous lecture) as well.**\n",
    "\n",
    "### Imputation important\n",
    "\n",
    "Imputation should be done over the training set, and then propagated to the test set. This means that the random sampling of categories should be done from the training set, and used to replace NA both in train and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((1022, 3), (438, 3))"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's separate into training and testing set\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(data[['BsmtQual', 'FireplaceQu', 'GarageType']],\n",
    "                                                    data.SalePrice, test_size=0.3,\n",
    "                                                    random_state=0)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let's create a variable to replace NA with a random sample of labels available within that variable\n",
    "# in the same function we create the additional variable to indicate missingness\n",
    "\n",
    "# make sure you understand every line of code.\n",
    "# If unsure, run them separately in a cell in the notebook until you familiarise with the output\n",
    "# of each line\n",
    "\n",
    "def impute_na(df_train, df_test, variable):\n",
    "    # add additional variable to indicate missingness\n",
    "    df_train[variable+'_NA'] = np.where(df_train[variable].isnull(), 1, 0)\n",
    "    df_test[variable+'_NA'] = np.where(df_test[variable].isnull(), 1, 0)\n",
    "    \n",
    "    # random sampling\n",
    "    df_train[variable+'_random'] = df_train[variable]\n",
    "    df_test[variable+'_random'] = df_test[variable]\n",
    "    \n",
    "    # extract the random sample to fill the na\n",
    "    random_sample_train = df_train[variable].dropna().sample(df_train[variable].isnull().sum(), random_state=0)\n",
    "    random_sample_test = df_train[variable].dropna().sample(df_test[variable].isnull().sum(), random_state=0)\n",
    "    \n",
    "    # pandas needs to have the same index in order to merge datasets\n",
    "    random_sample_train.index = df_train[df_train[variable].isnull()].index\n",
    "    random_sample_test.index = df_test[df_test[variable].isnull()].index\n",
    "    \n",
    "    df_train.loc[df_train[variable].isnull(), variable+'_random'] = random_sample_train\n",
    "    df_test.loc[df_test[variable].isnull(), variable+'_random'] = random_sample_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# and let's replace the NA\n",
    "for variable in ['BsmtQual', 'FireplaceQu', 'GarageType',]:\n",
    "    impute_na(X_train, X_test, variable)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "BsmtQual               24\n",
       "FireplaceQu           478\n",
       "GarageType             54\n",
       "BsmtQual_NA             0\n",
       "BsmtQual_random         0\n",
       "FireplaceQu_NA          0\n",
       "FireplaceQu_random      0\n",
       "GarageType_NA           0\n",
       "GarageType_random       0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# let's inspect that NA were replaced\n",
    "X_train.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual</th>\n",
       "      <th>FireplaceQu</th>\n",
       "      <th>GarageType</th>\n",
       "      <th>BsmtQual_NA</th>\n",
       "      <th>BsmtQual_random</th>\n",
       "      <th>FireplaceQu_NA</th>\n",
       "      <th>FireplaceQu_random</th>\n",
       "      <th>GarageType_NA</th>\n",
       "      <th>GarageType_random</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>64</th>\n",
       "      <td>Gd</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>0</td>\n",
       "      <td>Attchd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>682</th>\n",
       "      <td>Gd</td>\n",
       "      <td>Gd</td>\n",
       "      <td>Attchd</td>\n",
       "      <td>0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>0</td>\n",
       "      <td>Gd</td>\n",
       "      <td>0</td>\n",
       "      <td>Attchd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>960</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>TA</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>1</td>\n",
       "      <td>Attchd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1384</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Detchd</td>\n",
       "      <td>0</td>\n",
       "      <td>TA</td>\n",
       "      <td>1</td>\n",
       "      <td>TA</td>\n",
       "      <td>0</td>\n",
       "      <td>Detchd</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1100</th>\n",
       "      <td>TA</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Detchd</td>\n",
       "      <td>0</td>\n",
       "      <td>TA</td>\n",
       "      <td>1</td>\n",
       "      <td>Gd</td>\n",
       "      <td>0</td>\n",
       "      <td>Detchd</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     BsmtQual FireplaceQu GarageType  BsmtQual_NA BsmtQual_random  \\\n",
       "64         Gd         NaN     Attchd            0              Gd   \n",
       "682        Gd          Gd     Attchd            0              Gd   \n",
       "960        TA         NaN        NaN            0              TA   \n",
       "1384       TA         NaN     Detchd            0              TA   \n",
       "1100       TA         NaN     Detchd            0              TA   \n",
       "\n",
       "      FireplaceQu_NA FireplaceQu_random  GarageType_NA GarageType_random  \n",
       "64                 1                 Gd              0            Attchd  \n",
       "682                0                 Gd              0            Attchd  \n",
       "960                1                 TA              1            Attchd  \n",
       "1384               1                 TA              0            Detchd  \n",
       "1100               1                 Gd              0            Detchd  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>BsmtQual_NA</th>\n",
       "      <th>FireplaceQu_NA</th>\n",
       "      <th>GarageType_NA</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1022.000000</td>\n",
       "      <td>1022.000000</td>\n",
       "      <td>1022.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.023483</td>\n",
       "      <td>0.467710</td>\n",
       "      <td>0.052838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.151507</td>\n",
       "      <td>0.499201</td>\n",
       "      <td>0.223819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       BsmtQual_NA  FireplaceQu_NA  GarageType_NA\n",
       "count  1022.000000     1022.000000    1022.000000\n",
       "mean      0.023483        0.467710       0.052838\n",
       "std       0.151507        0.499201       0.223819\n",
       "min       0.000000        0.000000       0.000000\n",
       "25%       0.000000        0.000000       0.000000\n",
       "50%       0.000000        0.000000       0.000000\n",
       "75%       0.000000        1.000000       0.000000\n",
       "max       1.000000        1.000000       1.000000"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# let's transform the categories into numbers quick and dirty so we can use them in scikit-learn\n",
    "\n",
    "# the below function numbers the labels from 0 to n, n being the number of different labels \n",
    "#  within the variable\n",
    "\n",
    "for col in ['BsmtQual_random', 'FireplaceQu_random', 'GarageType_random',]:\n",
    "    labels_dict = {k:i for i, k in enumerate(X_train[col].unique(), 0)}\n",
    "    X_train.loc[:, col] = X_train.loc[:, col].map(labels_dict )\n",
    "    X_test.loc[:, col] = X_test.loc[:, col].map(labels_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Linear Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Test set random imputation\n",
      "Linear Regression mse: 6456070592.706035\n",
      "\n",
      "Test set random imputation + additional variable indicating missingness\n",
      "Linear Regression mse: 4911877327.956806\n"
     ]
    }
   ],
   "source": [
    "# Let's evaluate the performance of Linear Regression\n",
    "\n",
    "# first we build a model using ONLY the variable with the NA replaced by a random sample\n",
    "linreg = LinearRegression()\n",
    "linreg.fit(X_train[['BsmtQual_random', 'FireplaceQu_random', 'GarageType_random']], y_train)\n",
    "print('Test set random imputation')\n",
    "pred = linreg.predict(X_test[['BsmtQual_random', 'FireplaceQu_random', 'GarageType_random']])\n",
    "print('Linear Regression mse: {}'.format(mean_squared_error(y_test, pred)))\n",
    "print()\n",
    "\n",
    "# second we build a model including the variable that indicates missingness as well\n",
    "linreg = LinearRegression()\n",
    "linreg.fit(X_train[['BsmtQual_random', 'FireplaceQu_random', 'GarageType_random',\n",
    "                   'BsmtQual_NA', 'FireplaceQu_NA', 'GarageType_NA']], y_train)\n",
    "print('Test set random imputation + additional variable indicating missingness')\n",
    "pred = linreg.predict(X_test[['BsmtQual_random', 'FireplaceQu_random', 'GarageType_random',\n",
    "                             'BsmtQual_NA', 'FireplaceQu_NA', 'GarageType_NA']])\n",
    "print('Linear Regression mse: {}'.format(mean_squared_error(y_test, pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazing, in this exercise we can see the power of creating that additional variable to capture missingness. The mse on the test set decreased dramatically when we included these variables that indicate that the observations contained missing values. That represents a savings of:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1544193265"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "6456070592-4911877327"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1 billion dollars!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "**That is all for this demonstration. I hope you enjoyed the notebook, and see you in the next one.**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
